This article covers additional use cases that simplify the remote operation and control of the Expeditor integrator.

Transfer of large files in multiple messages

In addition to the underlying messaging system, Integrator can handle large resources at the application layer as well. At the messaging back-end, large resources can be split and transferred by multiple messages in order to get around limitations of the Expeditor micro broker and bridge. These resources must be reassembled by the Expeditor integrator appropriately. The Application Control Service (ACS) Activitiy MsgToFileActivity contains the algorithm for converting multiple messages to files.

The Queue Dispatcher Service recognizes messages that carry split resources in their payloads by the set sequence numbers and the end of data identifier. The Queue Dispatcher can also be configured to use a timer value to build a single target resource out of multiple messages (e.g. assemble all messages of the same type which were received within the last 10 minutes and write them into a single target file). The Queue Dispatcher Service puts these messages into a common sequence by setting the SequenceNumber and the EoD (End-of-Data) message header properties appropriately.

For file transfer, there are three possible scenarios to consider when messages are sent to the Expeditor integrator (i.e. they arrive in the “Msg2FTPQ”):

1. A file is sent as payload in a single JMS message where the payload is simply sent directly to the FTP server as a single file.
2. A file is sent as payload in a number of JMS messages within a given period of time where the message payloads are concatenated together in a common file when this time period has elapsed.
3. A file is sent as payload in a number of JMS messages delimited by an ”end of file” flag where the message payloads are concatenated together before sending to the FTP server when this flag appears.

The MsgToFileActivity only needs to recognize case 1) and case 3). For case 2), the Queue Dispatcher Service has already split the messages received within the given time interval (BuildInterval) into a sequence of messages that are clearly labeled by set SequenceNumbers and the EoD. The OSGi Event Admin event which triggers the corresponding flow is fired after the BuildInterval interval is over. This means that the MsgToFileActivity only needs to pull all messages that belong to one sequence of the corresponding queue.

As already discussed in previous chapters, the target file is created out of the following JMS custom message header properties:

• MessagePurpose and
• ResourceType
• SequenceNumber, EndofData and BuildInterval

These properties are the minimal settings for a received message carrying a file. Using the ResourceType as key, a default configuration for the target file can be retrieved from the Expeditor integrator’s local configuration. But, if ResourceType is not provided: TransferType, DestinationPath and DestinationName are required for determining the target file. Table 1 shows the algorithm for the file assembling out of multiple messages.

Table 1: Algorithm for assembling files from multiple messages

File Transfer Options	JMS Custom Message Header Properties			Comment
	SequenceNo	EndOfData	BuildInterval
Single file in a single message	0	true	0	Put payload of this message in corresponding target file (determined by JMS Custom Message properties and/or local XPDinteg Appl configuration).
Single file in multiple sequenced messages (of the same ResourceType and with a EOD indicator)	1	false	0	First message of a sequence of messages; collect all messages 1…n until EndOfData equals TRUE and write all payloads into a common file.
	<>1	true	0	Messages after first message out of a sequence of messages; collect all messages 1…n until EndOfData equals TRUE and write all payloads into a common file.
	<>1	false	0	Last message out of a sequence of messages; collect all messages 1…n until EndOfData equals true and write all payloads into a common file (determined by JMS Custom Message properties and/or local Expeditor integrator configuration.)

Note: If the JMS Custom Header property BuildInterval is set to ‘-1’, the configured default value in the XPDinteg.xml file is used:

Destination file naming convention and algorithms

For file transfer activities, the Application Control Service (ACS) Activity MsgToFileActivity is responsible for defining the name of the target file. The target file names are defined by the DestinationPath and the DestinationName properties that are either specified by the JMS custom headers or in the local Integrator configuration. The following specification applies for MessagePurpose=FileTransfer:

1. DestinationPath and DestinationName are optional JMS header properties and can be provided independently, for example:
   JMS-Header property values: DestinationPath=not set, DestinationName=VERKASS/ITEMRSF.TXT and Configured default settings in XPDinteg.xml: DestinationPath=C:/POS_DAT, default DestinationName=ITEMR.TXT become the destination file of C:\POS_DAT\VERKASS\ITEMRSF.TXT
2. DestinationName property values can be created dynamically by using operands to create the DestinationName:
   
   Table 2: Operands for dynamic file name creation
   

Operand	Description	Example
	creates a current time stamp as String in the format HHMMSS (from the original HH:MM:SS, but w/o delimiters)	DestinationPath=C:/POS_DAT; DestinationName =VERKASS/ITEMR_.TXT creates C:\\POS_DAT\VERKASS\ITEMR_140733.TXT
	current date stamp String in German format DDMMYYYY (from the original DD.MM.YYYY, but w/o delimiters)	DestinationPath=C:/POS_DAT; DestinationName =VERKASS/ITEMR_.TXT creates C:\\POS_DAT\VERKASS\ITEMR_03122007.TXT
	current date stamp String in en_US format YYYYDDMM (from the original YYYY-DD-MM., but w/o delimiters)	(equivalent to )
	last 8-digits of the TransactionId (EventId) from the JMS custom head	TransactionId=99912345678 and DestinationPath=C:/POS_DAT; DestinationName =VERKASS/.TIT create target resource C:\\POS_DAT\VERKASS\12345678.TIT If the TransactionId has less then 8 characters, the DestinationName will be created with leading zeros (e.g. TransactionId=1234 would lead to DestinationName=00001234.TIT)
	all digits of the TransactionId (EventId) from the JMS header	TransactionId=99912345678 and DestinationPath=C:/POS_DAT; DestinationName =VERKASS/ITEMR_-.TXT creates target resource C:\\POS_DAT\VERKASS\ITEMR_03122007-99912345678.TXT

Combinations of these operands are also valid, for example:


creates target file with the current date time in the name:

C:\\POS_DAT\VERKASS\ITEMR_03122007-140733.TXT

3. File name convention for BuildInterval<>0:

If target file is built out of a sequence of message (SequenceNo <>1 && BuildInterval=0) or the target file is filled with the content of all messages received within a defined BuildInterval interval, the target file name is set through the first message of the sequence (following the rules under 1.)

4. DestinationPath and DestinationName are created on the target system if they do not exist:

If the provided destination path and file name do not exist, they will be created. Insufficient access rights will cause the transaction to fail.

5. Example source and destination settings for retail PoS Systems:

• For PoS files from Back-end to the Store (BUC100, FTP):
  DestinationPath=/POS_DAT/VERKASS
• For Store Server files from Back-end to the Store (BUC102, Local File System):
  DestinationPath=C:/VERKASS
• For PoS files from Store to Back-end (BUC101, FTP):
  DestinationPath=/POS_DAT/ENTKASS
  (In this case the property DestinationPath contains the directory that is monitored for a given file.)

6. Destination file configuration examples:

• Configuration example for the ITEMR.TXT file (fixed file name):
  fixed default path = C:/POS_DAT/
  fixed default file name = ITEMR.TXT
  new: relative path name can be added: C:/POS_DAT/VERKASS/ITEMR.TXT
  PriceUpdFile[Description:Price Update
  File;TransportType:FTP;TransferMode:ASCII;DestinationCreationMode:APPEND;DestinationPath:/POS_DAT;DestinationName:VERKASS/ITEMR.TXT;Encoding:UTF-8;ftpServer:;ftpServerPort:21;ftpPassive:true;ftpUser:1234;ftpPassword:;ftpType:4690;queueType:FTP]
• Configuration example for the ITEMR.TXT file (dynamic file name):
  fixed default path = C:/POS_DAT/
  dynamic default file name = VERKASS/.TIT
  PriceUpdFile[Description:Price Update
  File;TransportType:FTP;TransferMode:ASCII;DestinationCreationMode:APPEND;DestinationPath:/POS_DAT;DestinationName:VERKASS/.TIT;Encoding:UTF-8;ftpServer:;ftpServerPort:21;ftpPassive:true;ftpUser:1234;ftpPassword:;ftpType:4690;queueType:FTP]
• Default configuration for BUC100:
  PriceUpdateFile (ITEMR.TXT) default config in XPDinteg.xml
  PriceUpdFile[Description:Price Update
  File;TransportType:FTP;TransferMode:ASCII;DestinationCreationMode:APPEND;DestinationPath:/POS_DAT;DestinationName:VERKASS/.TIT ;Encoding:UTF-8;ftpServer:;ftpServerPort:21;ftpPassive:true;ftpUser:1234;ftpPassword:;ftpType:4690;queueType:FTP]

Add footer to generated target files

Messages can carry content for a given target file. Under certain circumstances, it might be not sufficient to just detach the message payload as target file. The file content might need formatting or it might be required to insert additional data (for example, add file footer with dynamically created information).

For example, the 4690 PoS Controller requires a defined file footer line in order to process Price Update Files. This can be addressed by creating File Preprocessing Activities which are processed before the created target file is transferred from the Store Server to the 4690 Controller.

The BuildPriceUpdateFileFooterActivity in the buc100_PutPriceUpdate.flow addresses an example retail requirement for the Price Update File footer (see <XPDINTEG_HOME>/flowdefs ). The following file footer generation algorithm was implemented as reference:

With the variables $1-$7 representing the following values, the footer line will be built using the algorithm in Listing 16:

• $1- Date in YYMMDD format
• $2- Time in HHMM format
• $3- File Name (this will consume at least 12 characters, if the filename has only 5 characters, 7 white spaces must be added after it)
• $4- Line Count before applying the footer
• $5- Word Count before applying the footer
• $6- Byte Count before applying the footer
• $7- Byte by Byte algorithms (always set to 5 zeros)
• $8- Byte Count of first line (This is considered as the first line in the message payload which is interpreted as file. It is NOT the first line of the destination file if the destination file already exists!).

Listing 16: Footer line creation algorithm



Table 3: Example for retail price update file footer

Creation Mode	YYMMDD ($1)	HHMM ($2)	ENDE	ITEMR.TXT ($3)	LINE count ($4)	WORD count ($5)	BYTE count ($6)	BYTE-BY-BYTE ALGORITHM ($7)	BYTE count of 1st line in message ($8)
Sample value	070814	0953	ENDE	FILENAME	616	17422	130592	00000	128

Browse directory

The Expeditor integrator supports requesting a directory listing. A request message can be sent to the Expeditor integrator which use the provided JMS custom headers. This request message is an application control message and must be sent to the CtrlQ.

Table 4: Browse Directory message definition

MessagePurpose	TransportType	ResourceType	Required custom header properties and values / Comment
BrowseDirectory	FTP	Not set	-ResourceType is not used -DestinationPath -ResourceCmd (here: ftpServer:value1;ftpServerPort:value2;ftpPassive:value3;ftpUser:value4;ftpPassword:value5;ftpType:value6) -MessageId, TransactionId, LocationId, MessageSrcId
	LocalFileSystem		-ResourceType is not used -DestinationPath -MessageId, TransactionId, LocationId, MessageSrcId
	SSH		-ResourceType is not used -DestinationPath -ResourceCmd (here: sshServer:value1;sshServerPort:value2;sshUser:value4;sshPassword:value5;sshType:value6) -MessageId, TransactionId, LocationId, MessageSrcId

• The TransportType defines whether a local file system directory, a remote FTP or SSH directory must be accessed.
• The DestinationPath property contains the directory of which the file listing is requested.
• MessageId, TransactionId, LocationId and MessageSrcId must be provided so that the response message can be created appropriately.

The JMS Adapter (Resource Adapter) recognizes a new message and fires the corresponding default event with this topic:

• com/ibm/integrator/flowtriggerevent/BrowseDirectory/FTP/JmsAdapter or
• com/ibm/integrator/flowtriggerevent/BrowseDirectory/LocalFileSystem/JmsAdapter
• com/ibm/integrator/flowtriggerevent/BrowseDirectory/SSH/JmsAdapter

This event is received by the ACS which kicks off the configured default Browse Directory flow (in <XPDINTEG_HOME>/flowdefs/system ):

• Flow file name for FTP directory: Default_BrowseFtpDir.flow or
• Flow file name for local file system directory: Default_BrowseLocalFileSystemDir.flow
• Flow file name for local file system directory: Default_BrowseSshDir.flow

Depending on the TransportType, the triggered flow will contain either

• BrowseLocalFileSystemDirectoryActivity or
• BrowseFtpDirectoryActivity or
• BrowseSshDirectoryActivity

The created reply message will contain the properties and values according to Table 4.

Table 5: Reply message custom header properties and values for Browse Directory

Reply message custom header properties and values

Comments

ResourceType:not_set Encoding:UTF-8 TransferMode:ASCII MessagePurpose: BrowseDirectory TransportType:FTP | LocalFileSystem | SSH MessageId:__ TransactionId:{ | :__} EndOfData:true SequenceNo:1 LocationId:Store100 DestinationPath: MessagePriority: (JMS Message priority:MessagePriority) TimeStamp: MessageSrcId:

ResourceType is not used. TransactionId is set to the TransactionId of the request message. If this is not available, it is uniquely set according to Expeditor integrator ID schema. MessagePriority is set to the TransferPriority in the request message. If it is not set, the default value = 0 is used. The JMS message priority is set to the value of the MessagePriority custom header property (possible values: 0-9; [0]-lowest, [9]-highest).

The message body (payload) contains an XML structure with the names of the files in the directory of interest.

XML structure for Browse Directory reply message payload



where

• listingtype indicates the nature of the resource queried, e.g. ”FTP” or ”LocalFileSystem”.
• target indicates the target location queried i.e. a directory name.
• itemtype indicates an individual item type in the list, e.g. ”File” or ”Directory”.
• name indicates the name for the item in question, i.e. the file name.

Drain queues

The Expeditor integrator supports command messages that trigger the content deletion of local Expeditor integrator queues. A Drain Queue request message can be sent to the Expeditor integrator which use the provided custom JMS message headers. This request message is an application control message and must be sent to the Expeditor integrator CtrlQ. All messages that are stored in the requested local queue will be deleted.

Note: This is a very powerful command and must be handled with great care! Messages that were received within a transactional context might be deleted. A back-up copy of these messages is not available after deletion!

It is recommended to only use this command when remote Expeditor integrator runtime needs to be re-set, no back-end flows are running and previous Browse Queue command provided a list of messages that can be deleted. This command should only be used for house-keeping (e.g. delete content of DeadLetter queue regularly).

Table 6: Drain Queue message definition

MessagePurpose	TransportType	ResourceType	Required custom header properties and values / Comment
DrainQueue	-	-	-TransportType and ResourceType are not used -DestinationName (jndiDestinationKey of queue) -MessageId, TransactionId, LocationId, MessageSrcId

• The DestinationName property contains the jndiDestinationKey value (queue name) of the local queue of which the listing is requested.
• MessageId, TransactionId, LocationId and MessageSrcId must be provided so that the log messages can be created appropriately.

The JMS Adapter (Resource Adapter) recognizes a new message and fires the corresponding event with this topic:
com/ibm/integrator/flowtriggerevent/DrainQueue/MESSAGE/JmsAdapter

This event is received by the Application Control Service which kicks off the configured default Drain Queue flow (in /flowdefs/system):
Flow file name: Default_DrainQueue.flow
The triggered default flow will contain
DrainQueueActivity

Note: There is no corresponding reply message. Log messages are available for appropriate status information.

Browse queues

The Expeditor integrator supports requesting for a list of all messages in a local queue. This list contains the MessageIds of all messages that are currently stored in this queue. A Browse Queue request message can be sent to the Expeditor integrator which use the provided JMS custom headers. This request message is an application control message and must be sent to the Expeditor integrator CtrlQ.

Table 7: Browse Queue message definition

MessagePurpose	TransportType	ResourceType	Required custom header properties and values / Comment
BrowseQueue	-	-	-TransportType and ResourceType are not used -DestinationName (jndiDestinationKey of queue) -MessageId, TransactionId, LocationId, MessageSrcId

• The DestinationName property contains the jndiDestinationKey value (queue name) of the local queue of which the listing is requested.
• MessageId, TransactionId, LocationId and MessageSrcId must be provided so that the response message can be created appropriately.

The JMS Adapter (Resource Adapter) recognizes a new message and fires the corresponding event with this topic:
com/ibm/integrator/flowtriggerevent/BrowseQueue/MESSAGE/JmsAdapter

This event is received by the Application Control Service which kicks off the configured default Browse Queue flow (in /flowdefs/system):
Flow file name: Default_BrowseQueue.flow
The triggered default flow will contain
BrowseQueueActivity

The created reply message will contain the properties and values according to Table 4.

Table 8: Reply message custom header properties and values for Browse Queue

Reply message custom header properties and values

Comments

ResourceType: not set Encoding:UTF-8 TransferMode:ASCII MessagePurpose: BrowseQueue TransportType:Message MessageId:__ TransactionId:{ | :__} EndOfData:true SequenceNo:1 LocationId:Store100 DestinationName:< jndiDestinationKey of queue =DestinationName_of_reqMsg> MessagePriority: (JMS Message priority:MessagePriority) TimeStamp: MessageSrcId:

Message requests queue content listing (listing of message IDs) ResourceType is not used. TransactionId is set to the TransactionId of the request message. If this is not available, it is uniquely set according to Expeditor integrator ID schema. MessagePriority is set to the TransferPriority in the request message. If this is not set, the default value = 0 is used. The JMS message priority is set to the value of the MessagePriority custom header property (possible values: 0-9; [0]-lowest, [9]-highest).

The message body (payload) contains an XML structure with the MessageIds of the messages in the local queue of interest.

Listing 18 : XML structure for Browse Queue reply message payload

  <?xml version="1.0" encoding="utf-8" ?>

<listing type=”listingtype” targetName=”target”>

	<listing-item type=”itemtype” name=”name”/>

</listing>  

This is an unformatted list of all value-pairs of the JMS custom header properties of all messages in the queried queue where

• listingtype indicates the nature of the resource queried, here “message destination” .
• target indicates the target location queried i.e. jndi name of the Lotus Expeditor micro broker queue
• itemtype indicates the user defined headers of the message
• name indicates the value of the above headers in the message.

Execute Script

The Expeditor integrator supports control command messages that trigger the execution of script files that are provided in the message payload. A script file is transmitted in a message to the Expeditor integrator. The script file is detached to the configured (or set in the message header) local target directory and executed.

An Execute Script request message can be sent to the Expeditor integrator which use the provided JMS custom headers. This request message is an application control message and must be sent to the Expeditor integrator CtrlQ.

Note: This is a very powerful command and must be handled with great care! Script files run unattended outside Expeditor integrator’s control (not in transactional context) and can damage the whole target system!

It is recommended to only use this command if no other controlled alternative is available. This command must be specifically allowed in the Expeditor integrator configuration (set AllowScriptExecution=TRUE in XPDinteg.xml to allow script execution; it is set to FALSE by default).

Table 9: Execute Script message definition

MessagePurpose	TransportType	ResourceType	Required custom header properties and values / Comment
ExecuteScript	-	LocalFileSystemFile	-ResourceType is set to LocalFileSystemFile -DestinationPath & -Name -DestinationCreationMode (APPEND | OVERWRITE) -MessageId, TransactionId, LocationId, MessageSrcId

• The DestinationPath and -Name properties contain the target path in the local file system where the script file in the payload is detached to and is then executed. If no payload is provided and a file with the matching DestinationPath/-Name already exists, this will be executed instead.
• MessageId, TransactionId, LocationId and MessageSrcId must be provided so that the log message can be created appropriately.

The JMS Adapter (Resource Adapter) recognizes a new message in the CtrlQ and fires the corresponding event with this topic:
com/ibm/integrator/flowtriggerevent/ExecuteScript/LocalFileSystem/ScriptFile/JmsAdapter

This event is received by the Application Control Service which kicks off the configured default Execute Script flow (in /flowdefs/system):
Flow file name: Default_FileExecute.flow
The triggered default flow will contain
FileExecuteActivity

Example script file that can be transferred with the ExecuteScript control message.

 dir

ping 10.10.10.10

mkdir testdir 

The FileExecuiteActivity executes the commands and displays the results in the Expeditor integrator console. This activity can explicitly be further enabled/disabled by setting the context parameter EnableScriptExecution to TRUE or FALSE.

Note: There is no corresponding reply message. Log messages are available for appropriate status information.

Platform restart

The Expeditor integrator supports control messages that trigger stopping and re-starting of the complete runtime.

A Platform Restart request message can be sent to the Expeditor integrator which use the provided JMS custom headers. This request message is an application control message and must be sent to the Expeditor integrator CtrlQ.

Note: This is a very powerful command and must be handled with great care! Most of the all, the STOP operation should only be executed in emergency cases!
It is recommended to only use this command if no other controlled alternative is available. Furthermore, it is recommended to consider using a credentials check before the Platform Restart flow is executed (see details in the Installing and Configuring the IBM Lotus Expeditor integrator platform, see ‎Ref_1).

Table 19: Platform Restart message definition

MessagePurpose	TransportType	ResourceType	Required custom header properties and values / Comment
PlatformRestart	-	-	-ResourceCmd=Command:{restart|stop};Param:console,reset -MessageId, TransactionId, LocationId, MessageSrcId

• The ResourceCmd specifies the executed operation:
  ResourceCmd=Command:{restart|stop};Param:console,reset
  The Command specifies whether to restart or stop the Expeditor integrator runtime. The Param options further define whether:
  console – runtime runs in an open OSGi Console (recommended)
  reset –Expeditor integrator should be re-set before restarting (Careful: re-creates all local messaging components and deletes existing messages in queues!)
• MessageId, TransactionId, LocationId and MessageSrcId must be provided so that the log message can be created appropriately.

The JMS Adapter (Resource Adapter) recognizes a new message in the CtrlQ and fires the corresponding event with this topic:
com/ibm/integrator/flowtriggerevent/PlatformRestart/LocalFileSystem/JmsAdapter

This event is received by the Application Control Service which kicks off the configured default Platform Restart flow (in /flowdefs/system):
Flow file name: Default_PlatformRestart.flow

The triggered default flow will contain

	<XPDintegActivity 

		Name="PlatformRestart_PlatformRestart"

		ActivityName="XPDINTEG_PLATFORM_RESTART"

		LocalScriptFile="services/XPDintegRestart.bat"

		Command="restart"

		Param="console"

/> 

The PlatformRestart Activity executes the XPDintegRestart.bat script and displays the results in the Expeditor integrator console.

Note: There is no corresponding reply message. Log messages are available for appropriate status information.